Dense Text Retrieval based on Pretrained Language Models: A Survey

Text retrieval is a long-standing research topic on information seeking, where a system is required to return relevant information resources to user's queries in natural language. From classic retrieval methods to learning-based ranking functions, the underlying retrieval models have been continually evolved with the ever-lasting technical innovation. To design effective retrieval models, a key point lies in how to learn the text representation and model the relevance matching. The recent success of pretrained language models (PLMs) sheds light on developing more capable text retrieval approaches by leveraging the excellent modeling capacity of PLMs. With powerful PLMs, we can effectively learn the representations of queries and texts in the latent representation space, and further construct the semantic matching function between the dense vectors for relevance modeling. Such a retrieval approach is referred to as dense retrieval, since it employs dense vectors (a.k.a., embeddings) to represent the texts. Considering the rapid progress on dense retrieval, in this survey, we systematically review the recent advances on PLM-based dense retrieval. Different from previous surveys on dense retrieval, we take a new perspective to organize the related work by four major aspects, including architecture, training, indexing and integration, and summarize the mainstream techniques for each aspect. We thoroughly survey the literature, and include 300+ related reference papers on dense retrieval. To support our survey, we create a website for providing useful resources, and release a code repertory and toolkit for implementing dense retrieval models. This survey aims to provide a comprehensive, practical reference focused on the major progress for dense text retrieval

RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking

In various natural language processing tasks, passage retrieval and passage re-ranking are two key procedures in finding and ranking relevant information. Since both the two procedures contribute to the final performance, it is important to jointly optimize them in order to achieve mutual improvement. In this paper, we propose a novel joint training approach for dense passage retrieval and passage re-ranking. A major contribution is that we introduce the dynamic listwise distillation, where we design a unified listwise training approach for both the retriever and the re-ranker. During the dynamic distillation, the retriever and the re-ranker can be adaptively improved according to each other's relevance information. We also propose a hybrid data augmentation strategy to construct diverse training instances for listwise training approach. Extensive experiments show the effectiveness of our approach on both MSMARCO and Natural Questions datasets. Our code is available at https://github.com/PaddlePaddle/RocketQA.Comment: EMNLP 202

Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation

Knowledge-intensive tasks (e.g., open-domain question answering (QA)) require a substantial amount of factual knowledge and often rely on external information for assistance. Recently, large language models (LLMs) (e.g., ChatGPT), have demonstrated impressive prowess in solving a wide range of tasks with world knowledge, including knowledge-intensive tasks. However, it remains unclear how well LLMs are able to perceive their factual knowledge boundaries, particularly how they behave when incorporating retrieval augmentation. In this study, we present an initial analysis of the factual knowledge boundaries of LLMs and how retrieval augmentation affects LLMs on open-domain QA. Specially, we focus on three primary research questions and analyze them by examining QA performance, priori judgement and posteriori judgement of LLMs. We show evidence that LLMs possess unwavering confidence in their capabilities to respond to questions and the accuracy of their responses. Furthermore, retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries, thereby improving their judgemental abilities. Additionally, we also find that LLMs have a propensity to rely on the provided retrieval results when formulating answers, while the quality of these results significantly impacts their reliance. The code to reproduce this work is available at https://github.com/RUCAIBox/LLM-Knowledge-Boundary

A Survey of Large Language Models

Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions.Comment: ongoing work; 51 page

Trend in the study of intercultural competence: bibliometric analysis by citespace

The literature search has shown that the number of research materials associated with intercultural competence has been increasing gradually over the past twenty years. And this apparently shows that a systematic review of intercultural competence is essential for researchers and the aim of this paper is to discuss the trend of transition in intercultural communication studies from the notion of “intercultural competence”, followed by “intercultural communication competence” and the latest notion is “intercultural communicative competence”. In doing so, the Scopus database was adopted, by which the analysis was carried out on the keywords and their diachronic development with retrieval of materials published with the keywords used were “intercultural communication competence” or “intercultural communicative competence” by periodicals from 1970 to 2021. The findings show that the acronym ICC is used to represent all the three notions. Most importantly, the three representative notions emerge in different time but with continuous trait in the study of intercultural communication. Further, the main research aspects are identified in the domain of “education”, involving in “higher education”, “student”, and “human experiment” as well as “study abroad”. The findings of bibliometric analysis provide a more specific insights into the term selection of intercultural communicative competence, its main research aspects, diachronic development and future research direction

Directional scale elasticity considering the management preference of decision-makers

Most data envelopment analysis (DEA) studies on scale elasticity (SE) and returns to scale (RTS) of efficient units arise from the traditional definitions of them in economics, which is based on measuring radial changes in outputs caused by the simultaneous change in all inputs. In actual multiple inputs/outputs activities, the goals of expanding inputs are not only to obtain increases in outputs, but also to expect the proportions of such increases consistent with the management preference of decision-makers. However, the management preference is usually not radial changes in outputs. With the latter goal into consideration, this paper proposes the directional SE and RTS in a general formula for multi-output activities, and offers a DEA-based model for the formula of directional SE at any point on the DEA frontier, which is straightforward and requires no simplifying assumptions. Finally, the empirical part employs the data of 16 basic research institutions in Chinese Academy of Sciences (CAS) to illustrate the superiority of the proposed theories and methods

Joint multicast beamforming and relay design for maritime communication systems

With growing human maritime activities, supporting low-cost and high-speed information services for users at sea has become imperative. However, traditional means of maritime communications fail to provide high rate services due to their high cost and limited bandwidth. In this paper, considering abase station ashore and several offshore relay nodes, we propose a cooperative multicast communication scheme for maritime users relying on joint beamforming (BF) optimization and relay design. Specifically, we decompose our proposed joint optimization problem into two subproblems, which can be solved by the feasible point pursuit successive convex approximation approach. Furthermore, an alternating optimization algorithm is proposed, which imposes an exponentially increasing complexity as a function of the number of BF elements and the number of relays, when aiming for finding the globally optimal solution. In order to reduce this excessive computational complexity, a low-complexity distributed algorithm is also conceived and its closed-form solution is derived. Finally, the simulation results provided show that our proposed algorithm is beneficial in terms of increasing both the throughputas well as the energy efficiency

The transmit-energy vs computation-delay trade-off in gateway-selection for heterogenous cloud aided multi-UAV systems

Unmmaned Aerial Vehicles (UAVs) have been widely used in a range of compelling applications. In this paper, we integrate both the networking techniques and cloud computing tasks of multi-UAV systems. We commence by proposing an energy efficient scheme for selecting the gateway of UAVs invoked for relaying data to the heterogenous cloud. Then, relying on queuing theory and Lyapunov optimization, we strike a power-delay trade-off by jointly optimizing the computational task scheduling and resource allocation in the heterogeneous cloud architecture, which is comprised of an edge cloud and a powerful remote cloud. We analyse the optimal resource-allocation strategy for each time-slot and an iterative algorithm is conceived for reducing the computational complexity. Finally, our numerical results demonstrate the superiority of the proposed scheme